go之我怎样才能使这个脚本多线程

mfryf 阅读:71 2025-06-02 22:19:02 评论:0

我使用代码的方式是从文件中传递一个 url 列表并用我的代码运行每个 url,代码使用 bufio,所以我可以从文件中传递 url。

命令:猫 test2.txt | ./mygofile

https://beap.adss.yahoo.com/ 
https://id.answers.yahoo.com/search 
https://brokenurl 
https://id.answers.yahoo.com/KnowledgeSearchService 
https://id.answers.yahoo.com/question/nextQuestion 

示例 URL 的内容
package main 
 
import ( 
    "log" 
    "bufio" 
    "bytes" 
    "fmt" 
    "net/http" 
    "os" 
    "strings" 
    "crypto/tls" 
) 
 
func main() { 
    tr := http.DefaultTransport.(*http.Transport).Clone() 
    tr.TLSClientConfig = &tls.Config{InsecureSkipVerify: true} 
    client := &http.Client{Transport: tr} 
 
    passedUrl := bufio.NewScanner(os.Stdin) 
    var pUrl string 
    for passedUrl.Scan() { 
        fmt.Println(passedUrl.Text()) 
        if strings.Contains(passedUrl.Text(), "://"){ 
            pUrl = passedUrl.Text() 
        } else { 
            pUrl = "https://" + passedUrl.Text() 
        } 
 
        body := "<?xml version=\"1.0\"?>" 
 
        req, err := http.NewRequest("POST", pUrl, bytes.NewBuffer([]byte(body))) 
        if err != nil { 
            log.Println(err) 
        } 
 
        req.Header.Add("Content-Type", "application/xml; charset=utf-8") 
        resp, err := client.Do(req) 
        if err != nil { 
            log.Println(err) 
            continue // stop here and process next item 
        } 
 
        defer resp.Body.Close() 
 
    } 
} 

问题是当我提供大量 url 时代码非常慢。有什么方法可以让代码以简单的方式多线程或更快

请您参考如下方法:

创建 worker goroutine 池。 worker goroutine 从 channel 获取要处理的 url,并在 channel 关闭时退出。主 goroutine 将 url 提供给 channel 并在完成后关闭 channel 。

tr := http.DefaultTransport.(*http.Transport).Clone() 
tr.TLSClientConfig = &tls.Config{InsecureSkipVerify: true} 
client := &http.Client{Transport: tr} 
 
var wg sync.WaitGroup 
work := make(chan string) 
for i := 0; i < 10; i++ { 
    wg.Add() 
    go func() { 
        defer wg.Done() 
        for pUrl := range work { 
            fmt.Println(purl) 
            if !strings.Contains(purl, "://") { 
                pUrl = "https://" + passedUrl.Text() 
            } 
 
            body := "<?xml version=\"1.0\"?>" 
 
            req, err := http.NewRequest("POST", pUrl, strings.NewReader(body)) 
            if err != nil { 
                log.Println(err) 
                continue 
            } 
 
            req.Header.Add("Content-Type", "application/xml; charset=utf-8") 
            resp, err := client.Do(req) 
            if err != nil { 
                log.Println(err) 
                continue // stop here and process next item 
            } 
 
            // do something with resp 
 
            resp.Body.Close() 
 
        } 
    }() 
} 
 
passedUrl := bufio.NewScanner(os.Stdin) 
for passedUrl.Scan() { 
    work <- passedUrl.Text() 
} 
close(work) // signal workers to exit 
wg.Wait()   // wait for workers to exit 


标签:多线程
声明

1.本站遵循行业规范,任何转载的稿件都会明确标注作者和来源;2.本站的原创文章,请转载时务必注明文章作者和来源,不尊重原创的行为我们将追究责任;3.作者投稿可能会经我们编辑修改或补充。

关注我们

一个IT知识分享的公众号